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EXACT CALCULATIONS FOR FALSE DISCOVERY PROPORTION WITH 
APPLICATION TO LEAST FAVORABLE CONFIGURATIONS 

By Etienne Roquain and Fanny Villers 
UPMC University of Paris 6 

In a context of multiple hypothesis testing, we provide several 
new exact calculations related to the false discovery proportion (FDP) 
of step-up and step-down procedures. For step-up procedures, we 
show that the number of erroneous rejections conditionally on the re- 
jection number is simply a binomial variable, which leads to explicit 
computations of the c.d.f., the s-th moment and the mean of the 
FDP, the latter corresponding to the false discovery rate (FDR). For 
step-down procedures, we derive what is to our knowledge the first 
explicit formula for the FDR valid for any alternative c.d.f. of the 
p- values. We also derive explicit computations of the power for both 
step-up and step-down procedures. These formulas are "explicit" in 
the sense that they only involve the parameters of the model and the 
c.d.f. of the order statistics of i.i.d. uniform variables. The p-values 
are assumed either independent or coming from an equicorrelated 
multivariate normal model and an additional mixture model for the 
true/false hypotheses is used. This new approach is used to investi- 
gate new results which are of interest in their own right, related to 
least/most favorable configurations for the FDR and the variance of 
the FDP. 

1. Introduction. When testing simultaneously m null hypotheses, the false discovery pro- 
portion (FDP) is defined as the proportion of errors among all the rejected hypotheses and 
the false discovery rate (FDR) is defined as the average of the FDP. Since its introduction by 
Benjamini and Hochberg (1995) [H, the FDR has become a widely used type I error criterion, 
because it is adaptive to the number of rejected hypotheses. However, the randomness of the 
denominator in the FDP expression makes the study of the FDP and of the FDR mathematically 
challenging. 

There is a considerable number of papers that deal with the FDR control under different 
dependency conditions between the p- values (see for instance [l|, EM, 22])- In the latter, the 



goal is, given a prespecified level a, to provide a procedure with a FDR smaller than a (for any 
value of the data law in a given distribution subspace, e.g. for some dependency assumptions). 
For instance, the famous linear step-up procedure (LSU), also called the Benjamini-Hochberg 



proce 

procedure [l| (based on the Simes's line |29|). has been proved to control the FDR under inde- 
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pendence and under a positive dependence assumption (see [H, 0]). While controlling the FDR, 
one wants to maximize the power of the procedure, the power being generally defined as the 
averaged number of correct rejections divided by the number of false hypotheses. 

In this paper, we deal with the "reversed" approach: given the procedure, we aim to compute 
the corresponding FDR, or more generally the s-th moment and the c.d.f. of the FDP, and 
the Power. For procedures using a constant thresholding and under a mixture model assuming 
independence between the p-values, Storey (2003) addressed theses questions, while intro- 
ducing the positive false discovery rate (pFDR) (see afso Chi and Tan 0) . Considering step-up 
or step-down methods requires more efforts: when the p-values are all i.i.d. uniform, the exact 
distribution of the rejection number has been computed by Finner and Roters (2002) for step-up 
and step-down procedures, which leads to a computation of the FDR in the degenerate case 
where all the hypotheses are true When the p- values follow the particular "Dirac-uniform" 
configuration, that is, when the p-values associated to false hypotheses are equal to and when 
the p-values associated to true hypotheses are i.i.d. uniform, the FDP distribution has been 
computed by Dickhaus (2008) for general step-up-down procedures (see Section 3.7 in [8]). For 
an arbitrary distribution of the p- values under the alternative, Ferreira and Zwinderman (2006) 
gave a first exact expression for the moment of the FDP of the LS U p rocedure under indepen- 
dence 12]. Together with other recent approaches (see, e.g. 0, 24, 25, 27]), this puts forward a 



connection between the FDR expression and the rejection number distribution in the step-up 
case, under independence of the p-values. Additionally, Sarkar (2002) found an exact formula 
for the FDR, which is valid for any step-up-down procedure [26j]. However, it involves the c.d.f. 
of ordered components of dependent variables, by contrast with 0, and present paper, using 
c.d.f. of ordered components of i.i.d. uniform variables, so obtaining substantially more explicit 
formulas. 

Meanwhile, some of these approaches have also been investigated from the asymptotic point of 
view, when the number of hypotheses m tends to infinity; Chi (2007) computed the asymptotic 
rejection number distribution of the LSU procedure by introducing the criticality phenomenon 
[6J], while Finner et al. (2007) computed the limiting FDR of the LSU procedure for positively 
correlated p-values (following an equicorrelated multivariate normal model) in the particular 
Dirac-uniform configuration [141 ] . In this paper, the point of view will be mainly nonasymptotic. 

The new contributions of the present paper are as follows: 

• For a step- up procedure using a threshold (tk)k, we proved that the distribution of the 
number of erroneous rejections conditionally on k rejections is a binomial variable of pa- 
rameters k and pFDR(ifc) = -irotk/G(tk), where no is the (averaged) proportion of true 
nulls and G is the c.d.f. of the p-values. This provides new explicit formulas for the c.d.f. 
of the FDP, the s-th moment of the FDP (providing a correction with respect to [l2| for 
s > 3) and for the FDR, for any alternative distribution of the p-values. We also give an 
expression for the power, which yields a considerably less complex calculation than in (lB |. 
see Section I3TT1 

• Considering a step-down procedure, a new explicit formula for the FDR and the power is 
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presented under any alternative distribution of the p- values. To our knowledge, this expres- 
sion is the first one that clearly relates the FDR to (joint) rejection number distribution in 
the case of a step-down procedure and that is valid under any alternative, see Section 13.11 

• All the previous results, valid under independence between the p-values, can be easily 
extended to the case where the p- value family follows an equicorrelated multivariate normal 
model, by using a simple modification, see Section 13.21 However, this requires the use of 
a nonnegative correlation. The case of a possibly negative correlation is considered when 
only two hypotheses are tested, see Section 13.31 

• Our formulas corroborate the classical multiple testing results while they give rise to several 
new results. The two main corollaries hold under independence and are as follows. First, 
in Section [4.1.11 for the linear step-down procedure, we prove that a p- value configuration 
maximizing the FDR, i.e. a least favorable configuration (LFC) for the FDR, is the Dirac- 
uniform configuration. Additionally, considering a general step-down procedure, we define 
a new condition on the threshold ensuring that the Dirac-uniform configuration is still a 
LFC. As discussed in Section I4.1.H this condition is different from the one of the step- 
up case. Second, we found an exact expression of the minimum and the maximum of the 
variance of the FDP of the LSU, these extrema being taken over some p- value configuration 
sets of interest. The latter allows to better understand the behavior of the FDP around 
the FDR. In particular, this puts forward that the convergence of the FDP towards the 
FDR is particularly slow in the sparse case, see Section [4.2i 

All our formulas are valid nonasymptotically, that is, they hold for each m > 2. As a counter- 
part, they inevitably have a general form that can appear somewhat complex at first sight. For 
instance, denoting by *$> m the c.d.f. of the order statistics of m i.i.d. uniform variables on (0, 1), 
the FDR formula for step-up procedures requires the computation of ^ m at a given point of M m 
(at most), while the FDR formula for step-down procedures requires the computation of Vl/ m at 
2m different points of W 1 (at most). However, let us underline what are to our opinion the two 
main interests of this exact approach: 

- For some model parameter configurations and after possible simplifications, the formu- 
las are usable for further theoretical studies (monotonicity with respect to a parameter, 
convergence when m tends to infinity,...), as in Theorem 14. 1 1 and Theorem 14.31 

- For m not too large (say m < 1000), these formulas can be fully computed numerically, e.g. 
plotting exact graphs. Thus, they avoid using cumbersome and less accurate simulation 
experiments (extensively used in multiple testing literature) , see for instance Section 14.1.21 

2. Preliminaries. 

2.1. Models for the p-value family. On a given probability space, we consider a finite set of 
m > 2 null hypotheses, tested by a family of m p-values p = (pi,i £ {1, ...,m}). In this paper, 
for simplicity, we skip somewhat the formal definition of p- values by defining directly a p-value 
model, that is, by specifying the joint distribution of p. 
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In what follows, we denote by F the set containing c.d.f.'s from [0,1] into [0,1] that are 
continuous and by i*b(i) = t the c.d.f. of the uniform distribution over [0, 1] (we restricted our 
attention to the case where Fo(t) = t only for simplicity, all our formulas will be valid for an 
arbitrary Fq £ F). 

Definition 2.1 (Conditional p-value models). • The p-value family p follows the con- 
ditional independent model with parameters H = (-Hj)i<j<m G {0, l} m and F x G F, that 
we denote by p ~ P(h Fi)> P = (P*'* e {)-■> ■■■, m }) is a family of mutually independent 
variables and for all i, 

' F ifH l = 
F 1 ifH t = l ■ 



Pi 



The p-value family p follows the conditional equicorrelated multivariate normal model (say 
for short, conditional EMN model) with parameters H = (i?i)i<j< m G {0, l} m , p G [—{m — 
l) -1 , 1], and /j, > 0, that we denote by p ~ P(h pn)> f or a ^ P* ~ ^G^i + M-^i)? where 
the vector (Xj)i<j< m is distributed as a W m -valued Gaussian vector with zero means and 
a covariance matrix having 1 on t/ie diagonal and p elsewhere and where $ denotes the 
standard Gaussian distribution tail, that is, &(z) = P [Z > z] for Z ~ A/"(0, 1). /n i/iai 
model, the marginal distributions of the p-values are thus given by: 



Pi 




ifHi = 
¥(¥ _1 (t)-/x) if Hi = I 



The two above models are said "conditional" because the distribution of the p-values are 
defined conditionally on the value of the parameter H = (i^i)i<j< m G {0, l} m . The latter 
determines which hypotheses are true or false: Hi = (resp. 1) if the i-th null hypothesis is 
true (resp. false). We then denote by TIq{H) := {i G {1, ...,m} | Hi = 0} the set corresponding 
to the true null hypotheses and by mo(H) := \Ho(H)\ its cardinal. Analogously, we define 
Ui(H) := {i G {l,...,m} | Hi = 1} and mi(ff) := |Hi(F)| =m-m (F). 

To each one of the above models, we associate the "unconditional" version in which we endow 
the parameter H with the prior distribution B(l — 7To)® m , making (-ffj)i<j< m G {0, l} m being 
a sequence of i.i.d. Bernoulli with parameter 1 — ttq- On an intuitive point of view, this means 
that each hypothesis is true with probability ttq, independently from the other hypotheses. We 
thus define the following models for p (or more precisely for (H, p)): 

Definition 2.2 (Unconditional p-value models). • The couple (H, p) follows the uncon- 
ditional independent model with parameters ttq G [0, 1] and F\ G F , that we denote 
by (H,p) ~ P(no Fi) */ H ~ ^(1 — vro)® m and the distribution of p conditionally to 
H is P( H Fl y that is, conditionally on H, p follows the conditional independent model 
with parameters H and F\. In that model, the p-values are i.i.d. with common c.d.f. 

G(t)=7T Fo(t) + (l-7ro)Fi(t). 
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The couple (H,p) follows the unconditional equicorrelated multivariate normal model (say 
for short, unconditional EMN model) with parameters ttq G [0, 1], p G [— (m — 1) , 1], and 
p > 0, that we denote by (H,p) ~ P(- KQ ^ P ^), if H ~ B(l — 7ro)® m and the distribution of p 
conditionally on H is P^j that is, conditionally on H, p follows the conditional EMN 
model with parameters H , p and p. 



An important point is that the quantities mo(H) and mi(H) are deterministic in the con- 
ditional models P J ,P N , while they become random in the unconditional models P ,P with 
mo(H) ~ B(m, ttq) and m\{H) ~ B(m, 1 — tto). 

The conditional independent model is one of the most standard p- value models and was for 
instance considered in the original paper of Benjamini andHochberg (1995) 0|. Its unconditional 
version, also called the "random effects model", is very convenient and has been widely used 
since its introduction by Efron et al. (2001) [ill ], see for instance 

The conditional EMN model is a simple instance of model introducing dependencies between 
the p- values. It corresponds to a one-sided testing on the mean of Xi+pHi, simultaneously for all 
1 < i < m. It has become quite standard in recent FDR multiple testing literature; for instance, 
it was used in 14] with p, = oo and it has been considered in 0, 0] for numerical experiments. 



Furthermore, Efron (2009) recently showed that the EMN model may also be viewed as an 
approximation for some non-equicorrelated models, which reinforces its interest for a practical 
use [13] . In this model, provided that p > 0, the p- values are positively regression dependent 
on each one on the subset TIq(H) (PRDS on %q{H)) which is one dependency condition that 
suffices for FDR control (see [3[]). The unconditional version of this model is convenient because 
it provides exchangeable p- values (although not independent when p ^ 0). 

Additionally, we will sometimes consider the "Dirac-uniform configuration" for the above 
models. In that configuration, all the p-values corresponding to false nulls {Hi = 1) are equal to 
zero, that is, F\ is constantly equal to 1 for the independent models and p = oo for the EMN 



models. This configuration was introduced in 14J to increase the FDR as much as possible for 



the linear step-up procedures which thus appears as a "least favorable configuration" for the 
FDR (see also Section KM ■ 

2.2. Multiple testing procedures, FDP, FDR and power. A multiple testing procedure R is 
defined as an algorithm which, from the data, aims to reject part of the null hypotheses. Below, 
we will consider, as is usually the case, multiple testing procedures which can be written as 
a function of the p- value family p = (pi,i G {1, m}). More formally, we define a multiple 
testing procedure as a measurable function R, which takes as input a realization of the p-value 
family p G [0, l] m and which returns a subset R(p) of {1, m}, corresponding to the rejected 
hypotheses (i.e. i G R(p) means that the i-th hypothesis is rejected by R for the observed 
p- values p). 

Particular multiple testing procedures are step-up and step-down procedures. First define a 
threshold as any nondecreasing sequence t = (tfe)i<fc< m £ [0, l] m (with to = by convention). 
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Next, for any threshold t, the step-up procedure of threshold t, denoted here by SU(t), rejects the 
i-th hypothesis if pi < ty with k = maxjfc G {0, 1, m} \ pn,\ < where pm < pn) < ••• < 
P( m ) denote the ordered p- values (with the convention pm\ = 0). In particular, the procedure 
SU(t) using tk = ak/m corresponds to the standard linear step-up procedure of Benjamini and 
Hochberg (1995) denoted here by LSU. A less rejecting procedure uses a step-down algorithm; 
for any threshold t, the step-down procedure of threshold t, denoted here by SD(t), rejects the 
i-th hypothesis if pi < t^, with k = max{£; G {0, 1, ...,m} | V/c' < k, p^i^ < t/y}. Analogously to 
the step-up case, the procedure SD(t) using tk = ak/m is called the linear step-down procedure 
and is denoted here by LSD. 

Next, associated to any multiple testing procedure R and any configuration of true/false hy- 
potheses H G {0, l} m , we introduce the false discovery proportion (FDP) of R as the proportion 
of true hypotheses in the set of the rejected hypotheses, that is, 

(1) FDP (J?, H) = '^g^* 1 , 

where | • | denotes the cardinality function. Then, while for any multiple testing procedure R, the 
false discovery rate (FDR) is defined as the mean of the FDP (see [I] ) , the power is (generally) 
defined as the expected number of correctly rejected hypotheses divided by the number of false 
hypotheses. Of course, the FDR and the power depend on the distribution that generates the 
p- values, and we may use the models defined in Section 12.11 Formally, for any distribution P 
coming from a conditional model using parameter H G {0, l} m , we let 

(2) FDR(i?,P) = E p ^ P [FDP(i?(p),#)], 

(3) Pow(i?,P) = m 1 (H)- 1 M p „ P [\H l (H)nR(p)\]. 

Similarly, for any p-value distribution P coming from an unconditional model, the FDR and the 
Power use an additional averaging over H ~ B(l — iro)® m and are defined by: 

(4) FDR(i?,P) = E (H ^p[FBP(R(p),H)}, 

(5) Pow(R,P) = (7r 1 m)- 1 E Wp) ^[|-H 1 (P)ni?(p)|]. 

Remark that, for convenience, ([5]) is not exactly defined as the expectation of ([3|), because of the 
denominator. It corresponds precisely to the expected number of correctly rejected hypotheses 
divided by the expected number of false hypotheses. 

In the paper, to simplify the notation, we sometimes drop the explicit dependency in p, H 
or P, writing e.g. R instead of R(p), H instead of H (H), FDP(i?) instead of FDP(i?, H) and 
FDR(i?) instead of FDR(i?, P). 

2.3. Some notation and useful results. For any k > and any threshold t = (ii, ...,£&), we 
denote 

(6) *k(t) = **(ti ) ...,tfc) = P[l7 ( i ) <h,...,U (k) <t k ] . 
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where (JJj)\<i<k is a sequence of k variables i.i.d. uniform on (0, 1) and with the conven- 
tion \l/o(") = 1- In practice, quantity ([6]) can be evaluated using Bolshev's recursion ^(t) = 
1 " Eti ©(I " tjfe-i+i)%-i(*i, ...,t k -i) or Steck's recursion tf fe (t) = {t k f - Y^Zo (f)(*fc " 
tj + i) k ~ityj(ti, ...,tj) (see [28], p. 366-369). Additionally, the following relation holds (see Lemma 
2.1 in [15]): for all k G N and V\,v 2 £l such that < u\ + u 2 < v\ + kv 2 < L 

(7) tt fc (n + u 2 , i/ a + fci/ 2 ) = {vi + v 2 )(y x + (k + l)^)* -1 . 

From the ^fc's, we define the following useful quantities: for any threshold t = (tfc)i<fe<m and 
k > 0, k < m, we let 

(8) £>m(t,fc) 

(9) V m (t,k) 

Above, note that (tk) k and (1 — ifc+i) m ~ fc are correct when k = and k = m, even if 

is only defined for 1 < j < m. Note that Bolshev's recursion provides YlT=o ^m(t, k) = 

J2k=o'^ > m(i, k) = 1 for any threshold t. 

Finally, we will use the so-called Stirling numbers of the second kind, defined as coefficients {J} 
for s, £ > 1 by {q} = 0, {J} = for £ > s, {^} = 1 and the recurrence relation: for all 1 < £ < s+1, 
ft 1 } = + {ill}- For instance, {?} = 1, Q} = 3, {*} = 1, {}} = !,{*} = 7, {*} = 6, 
| 4 j = 1. From a combinatorial point of view, the coefficient j^j counts the number of ways to 
partition a set of s elements into £ (nonempty) subsets. The latter is useful to compute the s-th 
moment of a binomial distribution: if X ~ B(n,q), we have Vs > 1, E[A S ] = Yli=l ( n -i)! {i}^- 

3. New formulas. 



Vi-tfc+ir- fc * fc (ti,...,tfc). 



3.1. Unconditional independent model, m > 2. 



3.1.1. Step-up case. Let us consider the unconditional independent model. Finner and Roters 
(2002) derived the exact distribution of the rejection number of any step-up procedure in the 



case of i.i.d. uniform p- values (i.e., when all the hypotheses are true) 15j. In the unconditional 
model, the latter can be generalized as follows: denoting G(t) = iroFo(t) + (1 — iro)Fi(t) the 
common c.d.f. of the p-values, we have for < k < m that 

(10) P[|SU(t)| = fc] = V m ([G(t j )] 1 < j < m ,k). 



(this is straightforward from |15l ] because G is continuous increasing). 

Next, for the procedure R(t) = {i \ pi < t} using a constant threshold t 6 [0,1], Storey 
(2003) proved that the distribution of \7-Lq(H) n R(t)\ conditionally on |i?(i)| = k is a binomial 
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distribution B(k,iroFQ(t)/G(t)) (see proof of Theorem 1 in [301 ]. see also Proposition 2.1 in [3]). 
Later, Chi (2007) proved that the distribution of \Hq{H) n LSU| conditionally on |LSU| = k is 
asymptotically binomial (in a particular "supercritical" framework), see [f|. Here, we show that 
the latter holds non-asymptotically, for any step-up procedure, which, by using (fTUj) . gives exact 
formulas for the c.d.f. of the FDP, the s-th moment of the FDP, the FDR and the Power. 

Theorem 3.1. When testing m > 2 hypotheses, consider a step-up procedure SU(t) with 
threshold t and the notation of Section \2.3[ Then for any parameter ttq G [0, 1] and F\ € T , 
denoting G(t) = iroFo(t) +TViFi{t), we have under the generating distribution (H,p) ~ ■f > ( 7 r ,.F 1 ) 
of the unconditional independent model, for any k > 1, 

(11) \Ho(H) n SU(t)\ conditionally on \SU(t)\ = k ~ B{ k,^ ^' 



G(t k ) 



In particular, we derive the following formulas, for any x £ (0,1), for any s > 1, denoting by 
{^} the Stirling number of second kind and by [z\ the largest integer smaller than or equal to z, 

^ /ZA T?^+. \\ 3 El (+. \\ k ~i 



(12) n FDP(SU(t),H) < x] = EE (J) {^^)\^gW) ~'^ G Mte**»> k ) 

(13) E[^P(5C/(t), iT) s ] = E T^Tv J K E "4^ P m -4[G(t i+ ,)] 1 <,< m _,, k - t) 

m F (t ) 

(14) FDR(SU(t),P( n0)Fl )) = 7r m l<j<m-l,k- 1) 

fe=l 



(15) P ou ,(St/(t),P^ 0jFl) ) =E^i(^)^m-i([G , (t i+ i)]i< i < TO -i,A;-l). 



TO 



fc = l 



We can apply Theorem 13.11 in the case where t k = ak/m, to deduce the following results for 
the LSU procedure of Benjamini and Hochberg (1995) [l[: first, fflj]) leads to FDR(LSU) = Tr a, 
recovering the well known result of Benjamini and Yekutieli |3j in the unconditional model. 
Second, (fT5"j) provides the exact expression 

Pow(LSU) = E Fxiakfm) [™_ J (G(a/fc/m)) fc - 1 ^ m _ fc (l-G(am/m), 1-G(a(fc + l)/m)). 

Glueck et al. (2008) have obtained an exact expression for the power of the LSU under inde- 
pendence (in the conditional model) [3], but the corresponding formula was reported to have 
a complexity exponential in m, which is intractable for large m. Here, we obtained a much less 
complex formula, requiring (at most) the computation of the function ^ m at one point of R m . 
Third, formula (|13p used with tj. = ak/m is similar to expression (2.1) in Theorem 2.1 of [121 ]. in 
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which a formula for the s-th moment of the FDP of the LSU was investigated (in the conditional 
model). Our formula (|13p uses additional factors {^}. As soon as s > 3, they are definitely 
needed and they seem forgotten in for instance, taking a = 1, the corresponding linear 
step-up procedure rejects all the hypotheses and (fl3j) reduces to the computation of the s-th 
moment of a binomial distribution, which uses at least one {^} > 1, see Section [2.31 

Fourth, expression (|12p used with t k = ak/m yields what is to our knowledge the first exact 
expression for the c.d.f. of FDP (LSU), valid for any m > 2 and for any alternative c.d.f. F\. 

For instance, taking a typical Gaussian setting where F\{t) = (t) — we are able to 



evaluate the probability P(FDP(LSU) < c a) for c > 1; for \i = 3, a = 0.05, ttq = 1 — 1/y/m, 
expression 02} provides P(FDP(LSU) < a) ~ 0.724, P(FDP(LSU) < 2a) ~ 0.787 for m = 100 
and P(FDP(LSU) < a) ~ 0.557, P(FDP(LSU) < 2a) ~ 0.826 for m = 1000. This means that the 
LSU procedure, designed to control the FDR at level a, can have a FDP larger than 2a with a 
"non-negligible" probability, for some admittedly quite standard values of the model parameters. 
As m tends to infinity while the model parameters stay constant with m, the FDP converges to 



the FDR and the latter effect vanishes (see e.g. 2l|). However, we show in Section [4.21 that the 
convergence can be slow in the sparse case. 

Alternatively, some authors are interested in procedures R controlling the FDP, i.e. satisfying 



P[FDP( J R) < a] > 1 - 7, see e.g. [17j, l20J. As a matter of fact, by using {TT]), we directly 
deduce that the latter FDP control is satisfied by the step-up procedure SU(t*) using the oracle 
threshold defined by t* m+l = 1 (by convention) and for any 1 < k < m, 

t% = max{t e [0,t% +1 ] | F[X < ak] > 1 - 7 for X ~ B(k,ir Q t/G(t))}, 

with tt = if the above set is empty. However, the latter threshold is unknown in practice, 
because it depends on the c.d.f. G and an interesting issue is to estimate it. Chi and Tan (2008) 
introduced the threshold t% T = max{t £ [0, igjTj | F[X < ak] > 1 - 7 for X ~ B(k, 1 A 
(mt/k))} (with the convention t^_ 1 = 1). As a matter of fact, the latter can be seen as the 
empirical substitute of f£, because G(t?) ~ G m (t^) = k/m for any step-up procedure rejecting k 
hypotheses (G m denoting the e. c.d.f. of the p- values). Using the latter threshold, they established 
an asymptotic FDP control (as m tends to infinity). Here, a plausible explanation is that their 
procedure correctly mimics the oracle SU(t*) (asymptotically). 

3.1.2. Step-down case. In that section, we still consider the unconditional independent model, 
but we focus on the step-down case. By contrast with the step-up case, for a step-down procedure, 
the distribution of \Hq{H) nSD(t)| conditionally on |SD(t)| = k is not binomial, in general. For 
instance, we prove in Section [6.31 that for k > 1, k < m, 

(16) P[|ft (#)nSD(t)| = k I |SD(t) = it] =7r k^k{Fo(h),...,F {t k )) 



(17) W[\H (H) n SD(t)| = I |SD(t) = k] = tt\ 



* fc (G(ti) 



V k {G(h) 



,G(t k )) 



...,G(t k )) 
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and it turns out that a = b only for particular situations, such as t\ = ... = F\(x) = x or 
7To € {0,1}. Also, in the Dirac-uniform configuration F\ = 1 (an thus G(t) = not + tti), we 
establish in Section [6.31 that, for 1 < j < fc, 



(18) P[|^ (i?) n SD(t)| = j I |SD(t) = fe] 



k\ j k-j ^j(t k - j+1 ,...,t k ) 



jj *fc(7roti+7ri,...,7r ifc + 7r 1 J 



which is not binomial in general (see also Remark 13.31 below) . 

An exact expression for ¥[[Hq(H) n SD(t)| = j | |SD(t)| = k] only involving functions of 
the type and valid for any j, t and F\ seems considerably more difficult to derive. As a 
consequence, we now only focus on the calculation of the FDR and of the power. For this, let us 
recall the exact formula for the distribution of |SD(t)|: for all k 6 {0, ...,m}, 

(19) P[|SD(t)| =k] =V rn ([G(t j )] 1 < j < m ,k). 



(see formula (4) p. 344 of [28] and [15] which can be directly generalized in the unconditional 
independent model because G is continuous increasing). The next result, based on the calculation 
of the distribution of |SD(t')| conditionally on |SD(t)| = k (with t'j = connects the FDR 

to distributions of the type (fT9|) (see the proof in Section [6T2|) . 

Theorem 3.2. Form > 2 hypotheses, consider the unconditional independent model P(ir ,Fi) > 
a step-down procedure SD(t) with threshold t and the notation of Section \2.3[ Then for any pa- 
rameter ttq £ [0, 1] and F\ G T (denoting G(t) = ttqFq^) + niFi(t) ), we have 

m m , - 

FDR(SD(t),P{ mtFl) ) =7r mY J E -^ V m-i {(G(tj))i<j<m-i, k - l) 

k=l k'=k 

(20) *V m J( G ^t,? iM ) 



1 - G(t k ) 



l<j<m— k 



(21) Po«<523(t),P^ iJ?l) )=^Fi(t Jfe )C m _ 1 ([G(t i )] 1 ^< m _ 1 ,A;-l). 



k=l 



To the best of our knowledge, (|20j) is the first exact expression for a step-down procedure 
that relies the FDR to the (joint) distribution of rejection numbers, for any p-value alternative 
distribution. The main tool to get this result is Lemma IA.2I 

One straightforward consequence of ([20]) is that nomY^k=i ^k^m-i {[G(tj)]i<j< m -i, k — l) is 
an upper-bound of FDR(SD(t)), which in particular proves that FDR(SD(t)) is always smaller 
than FDR(SU(t)), as soon as t k /k is nondecreasing in k. While the latter result should probably 
be considered as well known, it is not trivial because when increasing the rejection number, both 
numerator and denominator are increasing within the FDP expression (for further developments 
on this issue, see Theorem 2 in [j^). 
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To illustrate (I20p . we may consider the case where tk = ak/m, that is, we may compute 
exactly the FDR of the LSD procedure. In the Dirac-uniform model F\ = 1 and using ([7]), we 
deduce the following expression: 



JJl III 7 / -1 \ / 7 

a" ( a\ \ -> \ - k I m — 1 \ I m — k 



2 mm 

7, — 1 „■ — 7, \ / \ J 



in—k 
n 



k=l j=k 

ofc\ *-2 / a(j _ fe + i) y-*-l / _ a (j + i) \ 



(22) X 7Q + ^o — 

\ m J \ m J \ m 

Finally, let us emphasize that expression (I20p is also useful to investigate least favorable config- 
urations for the FDR of step-down procedures (see Section [4, l.ip . 

Remark 3.3. If we only focus on the Dirac-uniform configuration, the FDP distribution 
of a given procedure R (rejecting any p-value equals to 0) only depends on the distribution of 
\RnT-lo(H)\ (conditionally on H), because \RnT-Lo(H)\ = \R\ —m\{H). As done in Section 3.7 
0/I1.A this leads to exact computations of the FDR for step-up, step-down and more general step- 
up-down procedures, in the particular Dirac-uniform configuration. In comparison, our approach 
is valid for an arbitrary alternative c.d.f. F\, while it intrinsically uses the exchangeability of the 
p-values (which requires to use an unconditional model). 

3.2. Unconditional EMN model with nonnegative correlation, m > 2. In this subsection, 
our goal is to obtain results similar to Theorem 13.11 and Theorem 13.21 but this time in the 
unconditional EMN model of parameters ttq, p and //, with a nonnegative correlation p G [0, 1]. 
In that case, we easily see that the joint distribution of the p-values can be realized as follows: 
for all i, pi = <&{yff> (U) + y/1 — p <& + pHi), where U, (Ui)i are all i.i.d. uniform 



on (0, 1) (and independent of (ifj)j). This idea can be traced back to Stuart (1958) [311 ] and 
Owen and Steck (1962) [22J. As a consequence, conditionally on U = u, the p-values follow the 
unconditional independent model of parameters ttq, Fq(- \ u, p) and F\{- \ u,p) where we let 

(23) F (t \u,p) = $ I y/T=p J ' ^ \u,p) = $ I ^ /T — I 

for p G [0, 1) and F (t | u, 1) = l{u < t}, Fi(t \ u, 1) = l{u < - p)} for p = 1. As a 

result, to obtain formulas valid in the unconditional EMN model, we may directly use formulas 
holding in the unconditional independent model (with the above modified c.d.f.'s) and using an 
additional integration over u G (0, 1). Hence, we deduce from Theorem 13.11 and Theorem 13.21 the 
following result (the formulas are not fully written for short): 

— N 

Corollary 3.4. For m > 2 hypotheses, consider the unconditional EMN model P^ 0tP ^ 
with parameters ttq G [0, 1], p, > and p G [0, 1] and let G(t\u, p) = ttqFq^u, p)+iriFi(t\u, p) us- 
ing notation (|23p . Then, for any threshold t, under the generating distribution (H, p) ~ P(ir ,p,ij,)> 
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the quantity F[FDP(SU(t)) < x] (resp. E[FDP(SU(t)) s }; FDR(SU(t)); Pow(SU(t))) is given by 
the RHS of (resp.§T5§; (HID; by replacing F (-) by F {- | u,p), -Fi(-) by F x (- | u, p) 

and G(-) by G(- \ u,p), and by integrating over u with respect to the Lebesgue measure on (0, 1). 
Additionally, a similar result holds for step-down procedures using (|20[) and (|2ip . 



In particular, applying Corollary l3.4l for the LSU procedure, we obtain that FDR(LSU, P(n 0iPilJ \) 
equals: 
(24) 

7TQ y~] ( , j / F (ak/m\u, p)G(ak/m\u, p) k ~ x ^ m ^ k {(l - G(a(m - j + l)/m\u, p))i<j< m -k)du. 
k=i \ / Jo 

An expression for lim m FDR(LSU, Pr m p „)) was provided by Finner et al. (2007), by considering 



the asymptotic framework where m tends to infinity JJj] . We compared the latter to the formula 
(|24p by plotting the graph corresponding to their Figure 3 (not reported here). The results are 
qualitatively the same for ttq < 1, but present major differences when ttq = 1 and p is small. This 



is in accordance with the simulations reported in the concluding remarks of Section 5 in [141 ] . 
Hence, the asymptotic analysis may not reflect what happens for a realistically finite to, which 
can be seen as a limitation with respect to our non-asymptotic approach. As illustration, when 
7To = 1, Finner et al. (2007) proved that lim p _>,o lim m FDR(LSU) = $(^—2 log a) < a whereas 
we have linip^o FDR(LSU) = a, as remarked in [14] using simulations and as proved formally 
in the next result. 

Corollary 3.5. For any to > 2 and for any threshold t, the quantities FDR(SU(t), Ptn 0>p>ll )) 
and FDR(SD(t), Pfa p fJ \) are continuous in any ttq G [0, 1], any p G [0, 1] and any p > 0. 

Corollary 13.51 is a straightforward consequence of Corollary 13.4) to prove the continuity in 
p = 1, we may remark that for any u outside the set S = {t^, 1 < k < to} U {<£(<!> (£&) — p), 1 < 
k < to} of zero Lebesgue measure, the functions Fo(t \ u,p) and Fi(t \ u,p) are continuous in 

In particular, Corollary 13.51 shows that the limit of the FDR when p tends to 1 is given by the 
degenerated case p = 1. In the latter case, the FDR is particularly easy to compute because only 
one Gaussian variable is effective: for step-up procedures, FDR(SU(t)) = irot m ] and for step- 
down procedures, FDR(SD(t)) = ZT=i (TK<~" min{t m „ fc+1 ,¥(¥~ 1 (t 1 ) - p)} (the proof is 
left to the reader). For instance, under the special p- value configuration where p = 1 and tvq = 1, 
the above FDR expressions yield FDR(SU(t)) = t m and FDR(SD(t)) = ti . Thus, as p ~ 1, the 
FDR value may considerably change as one considers a step-up or a step-down algorithm. 

Going back to Corollary 13.41 let us mention that the latter can be used in order to evaluate the 
FDR control robustness under Gaussian equicorrelated positive dependence for any procedure 
(step-up or step-down) that controls the FDR under independence. For instance, the adaptive 
procedures of Blanchard and Roquain (2008) [5] (step- up using t^ = a min{l, (1 — a)k/(m — k + 
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1)}) and Finner et al. (2009) [l3(] (step-up based upon t/. = ak/(m— (1 — ct)k)) have been proved 
to control the FDR at level a under independence (asymptotically for [HI). A simulation study 
was done in 0] in order to check if their respective FDR is still below a (or at least close to a) 
in the EMN model. Using our exact approach, we are able to reproduce their analysis without 
the errors due to the Monte-Carlo approximation. However, we underline that our approach uses 
non-random thresholds t; this is not always the case for adaptive procedures (see e.g. 0,0]). 



3.3. EMN model with a general correlation and m = 2. When the correlation p is nega- 
tive, the approach presented in the last section is not valid anymore and the problem seems 
considerably more difficult to tackle. We propose in this section to focus on the case where 
only two hypotheses are tested, which should hopefully give some hints concerning the behavior 
of the FDR under negative correlations for larger m. The next result follows from elementary 
integration and does not require the use of an unconditional model. 



Proposition 3.6. 
with parameters H - 



For m = 2 hypotheses, consider the conditional EMN model P^ H 
(Hi,H2) G {0, l} 2 (generating uiq G {1,2} true null hypotheses), p G 



[—1,1] and p > 0. Consider a threshold t = (ti 5*2)- Let z\ = $ *(i2) and Z2 = 1 (£].)■ Then 
FDR(SU(t),P { N Hpfi) ) is given by 



(25) 



P e (-1, l),m = l 
pe (-1,1), m = 2 



■1, mo = 1 



p = -l,TOo = 2 



1 f*(zi 

2 JO 



z\— p$ 1 (w) 



dw + h( Z1 -^ 
dw + J} $ 



Z2— Pfo (w) 



Zj—pfy 1 {w) 



dw 



dw 




if < p < 2zi 
z\) if 2z\ < p < z\ 
Zi) if p> zi+ z 2 
if 1/2 > £ 2 
t 2 )-l if 1/2 <t 2 ,ii +*2 < 1 
if 1/2 < t 2 ,t 1 +t 2 > 1 



Z2 



p = l,mo = 1 
p = l,m = 2 
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and FDR(SD(t) , P/^ %) is given fry 



(26) 



p e (-1,1), m = 1 
p € (-1,1), m = 2 


1 r«(*-M) ¥ / n-p*- 1 ^) \ ^ + 1 r*(*l-/*) $ f ) ^ 

2 JO I y/l-p 2 J 2J$( 22 - M ) y V 1 -^ / 


p = -l,m = 1 
p = — 1 , mo = 2 


< j(tl+*2)-3*(A*-*2) 
min(2ii, 1) 


*/ < fi < zi + z 2 
+ !$(// -zi) if zi + z 2 <p<2z 2 
*/ P > 2z 2 


p = l,m = 1 
p = l,mo = 2 


i 

2 
/ 


min(t 2 ,$(z 2 - M)) 

i 





4. Application to least/most favorable configurations. 

4.1. Least favorable configurations for the FDR. In order to study the FDR control, an 
interesting multiple testing issue is to determine which are the values of the model parameter 
Pi (or n) for which the FDR is maximum. The latter is called a least favorable configuration 
(LFC) for the FDR. 

4.1.1. Independent model. Let us focus on the unconditional independent model. For a step- 
up procedure, expression (fTl"]) can be seen as nornEolt^/k] (where k = |SU([G(tj+i)]i<j< m _i)| + l 
and Eo denotes the expectation with respect to m — 1 i.i.d. uniform p- values). This shows 
that the behavior of the function k t— >■ t^/k is crucial to determine the LFC's of the FDR. 
Namely, if F% and F{ are two c.d.f.'s such that for all t G [0,1], F\(t) < F[(t), then we 
have FDR(SU(t),P( 7r0) jr 1 )) < FDR(SU(t), P^ no>F ^) when tf./k is nondecr easing, while we have 

FDR(SU(t), P^ Fi ~j) > FDR(SU(t), P( wo ^f[)) when tk/k is nonincreasing. The latter recovers 
a well known result of Benjamini and Yekutieli (2001) which was initially established in the 
conditional model, see Theorem 5.3 in [jj. As a consequence, the LFC for the FDR is either 
Pi = 1 (Dirac-uniform) when t^/k is nondecreasing, or Pi = (F\{x) = x if we only look at 
concave c.d.f.'s) when t/~/k is nonincreasing. In the border case of a linear threshold, the FDR 
does not depend on Pi (e.g. it is equal to hqo. for the LSU), hence any configuration is a LFC. 

An open problem is to determine the LFC's of a step- down procedure using a given threshold 
t = (tj)i<j< m - Here, we introduce a new condition on the threshold t which provides that the 
Dirac-uniform configuration is a LFC for the FDR of the corresponding step-down procedure. 
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We define for any threshold t = (tj)i<j< m the following condition: 

(A) k G {l,...,m} i y ; ^ k T> m _u ( ( -^-^ | , i | is nondecr easing. 

We now present the main result of this section, which uses Theorem 13.21 and is proved in Sec- 
tion [631 



Theorem 4.1. Form > 2 hypotheses, consider the unconditional independent model Pi^ Fl ) 
and a step-down procedure SD(t) with a threshold t = (tj)i<j<m satisfying (pD) , Then for any 
ttq G [0, 1] and concave c.d.f. F\ G T , we have 

(27) FDR(SD(t),Pl (hFl) ) < FDR(SD(t),Pl (hFl=1) ), 

meaning that the Dirac-uniform distribution is a least favorable configuration for the FDR of 
SD(t). Moreover, for a G (0,1), the linear threshold t = (aj /m)i<j< m satisfies ([2]) and thus 
(|27p holds for the linear step-down procedure LSD. 



While condition (L4|) may be somehow difficult to state formally, it is very easy to check 
numerically because it only involves a finite set of real numbers. For instance, considering 
the threshold of Gavrilov et al. (2009) th = ak/{m + 1 — (1 — a)k) (for which the step- 
down procedure controls the FDR, see [la)), we may see that (tk)k satisfies ((31) for each 
(a,m) G {0.01,0.05,0.1,0.2,0.5,0.9} x {5,10,50,100}, for instance. In fact, we were not able 
to find a value of (a, m) for which the corresponding threshold does not satisfy (^4|) (unfortu- 
nately, we have yet no formal argument proving for any value of (a, m)). As a consequence, 
Theorem 14.11 states that the LFC for the procedure of Gavrilov et al. (2009) is still the Dirac- 
uniform configuration (over the class of concave c.d.f. 's), at least for the previously listed values 
of (a,m), which is a new interesting finding. 

In comparison with the step-up case, for which the standard condition u k \— > t^/k nonde- 
creasing" provides that the Dirac-uniform configuration is a LFC, the new sufficient condition 
(fAl) in the step-down case may be written as " k t— >■ %/k . *$>(t,k) is nondecreasing" , where 

*(t, k) = YT=Q k Y+prn-k ( ( tk t-t tk ) , i J . It turns out that the additional function * 

k / l<j<m— k J 

has a quite complex behavior, not necessarily connected to the one of t^/k, so that there is no 
general relation between ([21) and u k \— > t^/k nondecreasing"; for instance, on the one hand, ([21) 
does not hold for the piecewise linear threshold defined by t^ = apk/m for 1 < k < a and 
by ifc = a(pa — m)((o — m)m)~ 1 k + a(l — (pa — m)(a — m) _1 ) for a + 1 < k < m (using e.g. 
m = 50, p = 0.6, a = 4, a = 0.5), while t^jk is nondecreasing. On the other hand, ([2]) holds 
for t}~ = 0.9 (A:/m) 9 / 10 (using e.g. m = 50) while t^/k is decreasing. In particular, for the latter 
threshold and considering only the set of concave c.d.f.'s, a LFC for FDR(SU(t)) is F\(x) = x 
while a LFC for FDR(SD(t)) is F\ = 1 (and we checked numerically using Gaussian models 
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that F\(x) = x is not a LFC for FDR(SD(t)) ). This puts forward the complexity of the issue: 
whether we consider a step-up or a step-down procedure, the LFC's for the FDR may be different 
for some thresholds (e.g. tk = 0.9 (fc/m) 9//1 °, m = 50) and they may coincide for some other 
thresholds (e.g. the one of [16[ for suitable (a,m)). 

4.1.2. EMN model. When the p-values follow the EMN model, Finner et al. (2007) conjec- 
tured that a LFC for the FDR of the LSU is still the Dirac-uniform distribution (see Section 1 
in [l3]). Here, we support this conjecture when p > but we disprove it when p < 0. 

In order to investigate this issue, we reported on Figure [T] the FDR of the LSU procedure 
against p in the EMN model when p > (left) and when p < (right), by using Corollary 13 .41 and 
Proposition 13.61 Under positive correlation, although each curve is not necessarily nondecreasing 




Fig 1. FDR(LSU) against the mean p. Left: p > unconditional EMN model m — 100 and no = 0.5. Right: p < 
conditional EMN model with m = 2 and mo = 1. a = 0.05. 

(e.g. for p = 0.2), the case p = oo, close to the right most point of FigureHJ seems to be a LFC. 
A challenging problem is to state the latter formally. Under negative correlation and m = 2, 
however, p = oo is not a LFC anymore. As a matter of fact, in the case where m = 2, tuq = 1 
and p = oo, the two p- values are independent (one p- value equals 0), so that the FDR equals 
a/2 = amo/m which is not a maximum for the FDR, as we will show below. 

Qualitatively, we observed the same behavior concerning the FDR of the LSD procedure. 

Under negative correlation, the Dirac-uniform is not a LFC for the FDR and we can therefore 
legitimately ask what are the LFC's in that case. Here, we propose to solve this problem when 
m = 2 in the conditional EMN model. Let z\ = $ (a) and Z2 = $ (a/2) and first consider 
the LSU procedure. Its FDR is plotted in Figure [2] (top). When ttiq = 1, we can check that 
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(p, p) = (— l,zi + z-i) is a LFC because applying (|25l) . the corresponding FDR is 3a/4, which 
equals the Benjamini-Yekutieli's (BY) upper-bound (1 + 1/2 + ... + l/m)amo/m 0] (valid under 
any dependency). Interestingly, in general, Guo and Rao (2008) states that the BY bound 
can be fulfilled using very specific dependency structures between the p-values (not necessarily 
including those coming from a EMN model) [ijj] . Here, we remark that the maximum value 
of the FDR still equals the BY bound for mo = 1, even for p- values coming from a EMN 
model. Next, for mo = 2, we may differentiate the corresponding expression in (|25p in order to 
obtain that, assuming a < 1/2, the FDR (that does not depend on p) attains its maximum in 
p = (-^ - z 2 ) - {(zf - Zl z 2 f + 21og(2)(*2 - z\) + 41og(2)} 1 /2 )/(2 i og (2)) G (_l, ). 

Second, we consider the LSD procedure, whose FDR is plotted in Figured] (bottom). In the 
case where mo = 2, by differentiating the corresponding expression in (|25p (that does not depend 
on p), we are able to state that the FDR attains its maximum at p = — 1 and that the value of 
the maximum is a. In particular, the FDR of the LSD procedure is always smaller than or equal 
to a when m = 2, in the conditional EMN model (for any mo) and thus also in the unconditional 
EMN model, even for a negative correlation. An interesting open problem is to know whether 
this holds for larger m. 




° ^To ^0^5 Oft 0.5 1.0 ^0 ^5 ofo 0^5 TTcT 




-1.0 -0.5 0.0 0.5 1.0 -1.0 -0.5 0.0 0.5 1.0 

rho rho 



Fig 2. FDR against the covariance —1 < p < 1 in the conditional EMN model, m = 2; a = 0.2. Left: mo = 1, 
right: m = 2. Top: LSU, bottom: LSD. 

Remark 4.2. Reiner- Benaim (2007) also studied the value of p maximizing the FDR in the 
case m = 2 in the (conditional) EMN model with possibly negative correlation fjjj /. The latter 
work focused on the two-sided testing with p E { — 1, 1}, mo = 1 and m = 2. 
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4.2. Least/most favorable configurations for the variance of the FDP. We focus here on the 
unconditional independent model and on the LSU procedure. Using f)13|) with s = 2, we easily 
derive the following expression for the variance of the FDP: 



(28) V[FDP(LSU)] 



m j 



v 



m—l 



([^(j + ljHlK, 



<m— 1 ; 



l) - (a-K ) 2 /m. 



k=l 



As a consequence, by contrast with the FDR which is constantly equal to in that case, the 
variance of the FDP depends on the alternative p- value c.d.f. F\. Moreover, since the sum in 
(f28|) equals Eo[(|SU([G(a(j + l)/m)]i<j< m _i)| + l) -1 ] (where Eo denotes the expectation with 
respect to m — 1 i.i.d. uniform p- values), the smaller F\ (point-wise), the larger the variance. 
Therefore, over the set F\ G J 7 , the least favorable configuration for the variance (that is, the 
configuration where the variance is the largest) is given by F\ = while the most favorable 
configuration (that is, the configuration where the variance is the smallest) is the Dirac-uniform 
configuration F\ = 1. Over the more "realistic" c.d.f. sets F' = {F\ G F | Vx G (0, 1), F\(x) > x} 
and T e = \F\ G T \ Vx G (0, l),Fi(x) > e}, < e < 1 the least favorable configurations for the 
variance are given respectively by F\(x) = x and F\(x) = e. For these extreme configurations, 
the expression of the variance (|28p can be simplified by using the next formula (proved in 
Section f6.6H : for any threshold of the form tk = (3 + kj, 1 < k < m, with /3, 7 > 0, 



(29) 



E [(|SU(t)| + l 



1 



7-/3 



(1 + 7-/3) 



m+l 



1 



m + l 



7 [(l + 7-/3) m -l] 



for 7 ^ /3 and Eo[(|SU(t)| + 1) x ] = 1 — rwy otherwise (where Eo denotes the expectation with 
respect to m i.i.d. uniform p-values) This leads to the following result. 

Theorem 4.3. Consider the linear step-up procedure LSU in the unconditional independent 
model with parameters ttq and F\. Then for any m > 2, a G (0, 1), ttq G [0, 1] and e G (0, 1], 
under the generating distribution (H,p) ~ P^ 7To Fl ^, the following holds: 



min {Y[FDP(LSU)}} 



(30) 



max{V[FDP(LSlf)}} 
msx^YlFDP^SU)]} 
max {Y[FDP(LSU)\} 



mm {V[FDP(LSU)}} = mm Y[FDP(LSU)}} 



avro 1 - irff _ (a7T ) 2 
m 1 — ttq m 

a7To(l — Oi7To) 

a7r (l - a) + (1 - 7r ) 



7T, 



m—l 




1 - 7T 



7TQa 



771 



a7r 1 - (1 - (1 - 7r )e) 



77; 



(1 - vr )e 



777 



+ 1 



1-1 



;i - vr )e 



,m— 1 



(1 - 7T )£ 



+ 1 
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The proof is made in Section 16,61 Using Theorem 14.31 we are able to investigate the following 
asymptotic multiple testing issue: does the FDP converge to the FDR as m grows? Establishing 
the latter is crucial because when one establishes FDR < 7, one implicitly wants that for the 
observed realization oj, the control FDP(w) < 7' still holds for 7' ~ 7 (at least with high 
probability and when m is large). Here, the variance measure the L 2 distance between the FDP 
and the FDR, and since FDP 6 [0, 1] the latter distance tends to zero if and only if the FDP 
converges to the FDR in probability. First, if ttq £ (0, 1) does not depend on m, the convergence 
holds over the set of c.d.f.'s J- £ , with a distance (E(FDP — FDR) 2 ) 1 / 2 converging to zero at 
rate 1/y/m. This corroborates previous asymptotic studies in the so-called "subcritical" case 
(see e.g. 0, [2]]]). By contrast, when considering the classes T and T' the convergence does not 
hold in the least favorable configurations F\(x) = and F\{x) = x, respectively. The latter 
is quite intuitive because the denominator in the FDP does not converge to infinity anymore 
in that cases (see e.g. [3]), so these configurations can probably be considered as "marginal". 
Second, our non-asymptotic approach allows to make ttq depends on m in the following way 
1 — 7To = 1 — 7To >m ~ m~P with < /3 < 1, which corresponds to a classical "sparse" setting 
(see e.g. [9(). Expression (j30|) implies that, in this sparse case, the variance is always larger 
than a quantity of order l/m 1- ' 3 . In particular, when 1 — 7To im ~ 1/m, for any alternative c.d.f. 
jPi, the FDP does not converge to the FDR, and when 1 — ttq^ ~ m~P with < /3 < 1, 
for all F\, the convergence of the FDP towards the FDR is of order slower than I/ttt/ 1- ^/ 2 
(in I? norm). As illustration, for m = 10000, 1 — ttq = 1/100 and a = 0.05, expression (|30p 
gives (E(FDP - FDR) 2 ) 1 / 2 > 0.0217, so the FDP has a distribution quite spread around the 
FDR = iTQa ~ 0.05. As a conclusion, considering a sparse signal slows down the convergence of 
the FDP to the FDR, so any FDR control should be interpreted with cautious, even in this very 
standard framework (independent p- values with the LSU procedure). 

5. Extensions. Our approach is also useful to study the false non-discovery proportion 
(FNDP), that is, the proportion of false hypotheses among the non-rejected hypotheses, and 
in particular the false non-discovery rate (FNR), defined as the average of the FNDP. For 
this, we use the following duality property between step-up and step-down procedures: point- 
wise, the hypotheses rejected by SD(t,p) are exactly the hypotheses non-rejected by SU(t, p) 
with p i = 1 — pi and t r = 1 — t m _ r+ \. Hence the distribution of the FNDP of a step-down 
procedure can be deduced from the distribution of the FDP of a step- up procedure. Precisely, 
for < k < m— 1, the property (jlip implies that the distribution of the erroneous non-rejection 
number \Hi(H) n (SD(t)) c | conditionally on |SD(t)| = k is binomial with parameters m — k and 
7Ti(l — — G(tfc +1 )). In particular, this leads to 

m ~ 1 1 W (i \ 

FNR(SD(t)X 0iFi ) = 777.71" 1 ~ m _fc + - Pm-l((G(tj))i<j< m -i,fc)- 

Moreover, applying once more the duality property between step-up and step-down, we deduce 
from Section [3.1.21 that for a step- up procedure, the distribution of the erroneous non-rejection 
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number conditional on the rejection number is not binomial, in general, while we can still obtain 
an explicit expression for the FNR. 

Finally, since our formulas depend on the true parameters of the model, which are in general 
unknown in a statistical approach, one may formulate the concern of estimating these quantities 
in our formulas. We did not investigate in detail this issue as it would exceed the scope of this 
paper. Here, we simply notice that plugging convergent estimators of the parameters in our 
formulas will lead to convergent estimators for the corresponding quantity (e.g. P(FDP < x), 
FDR or Power), because our formulas are continuous in all the model parameters. 

6. Proofs. 

6.1. Proof of Theorem \ 3.1[ Let us first prove (|lip by computing the joint distribution of 
\Ho(H) n SU(t)| and |SU(t)|. In the independent unconditional model, we may use the ex- 
changeability of (Hi,pi)i to obtain for any < j < £ < m, 

p[|H (tf)nsu(t)|=j,|su(t)| = £] 

'A / TO 



; ) . l w (s)nsu(t) = {i,...,i},su(t) = {!,...,£}] 



£ ) (™V[SU(t) = {!,...,*},#! = ... = Hj = 0,H j+ i = ... = H t 



Next, by definition of a step-up procedure, we have SU(t) = {i \ ps < tt} with k = |SU(t)| 
(expression related to the "self-consistency" condition introduced in [J]). Using Lemma lA.H if 
k%-, denotes the number of rejections of the step-up procedure of threshold (ij + f)i<j< m _^ over 
m — I hypotheses and using the p- values pe+i, p m , we have 

SU(t) = {1, ...,£} ^ Pl < t t , ...,pt <k,k = t 

<s=^ pi < k, ...,pi < k, k[ e) = o. 

Therefore, 

p[|^ (^)nsu(t)| =j,|su(t)| =£] 

= r) (7) p ^ ^ '~>p* ^ hi Kti = °' Hi = - = B i = = - = H t = i \ 

x F[p J+1 < k, ...,Pt < t e , H j+1 = ... = H e = l] ¥[k[ e) = 0] 

(™\ (7r Fo(^)) j (7riFi(^)) £ -% m ^(l - G(t m ), 1 - G(t t+1 )) , 
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where we used the independence between the (Hi,pi) in the second equality. This leads to (fTTI) 
and then to (| 12 [) . For (|13p . we use the Stirling numbers of second kind and the formula of the 
s-th moment of a binomial distribution of Section 12.31 Expression (|14ft is a direct consequence 
of (fr3j) for s = 1. For the power computation, from ([TT]) . the distribution of l'Hi(il) n SU(t)| 
conditionally on fc = |SU(t)| is binomial with parameters fc and -KiF\{t^)/G{t^). Therefore, 
mHi(H) n SU(t)|] = E [7ri fciq (t^)/G{t^)] and (HSJ follows. 

6.2. Proof of Theorem \3.Sk Let us prove the FDR expression (the proof for the power is 
similar). Define k = |SD(t)| and fc(i), ki^ as in Lemma IA.2I We get by exchangeability of 
(Hi,pi)i and independence of the p- values, 



FDR(i?) = J^E 



i=i 
mE 



Hpj < t~ k } 

fc v 1 
l{pi < * 



= 0} 



mE 



i{pi < 



} 



*co + 1 



-!{#! = 0} 



7TomE 



V 1 

F o(*fc (1)+ i 



l{£Ti = 0} 



fc' (1) + 1 



Therefore, expression (f20|) will be proved as soon as we state that for any k, k' with < k < m 
and k < k' < m, that we have 



|SD(t)| = fc,|SD(t')| = k'] 



(31) 



2?m((G(£j))i<j< m , k) V 



m—k 



G(tk+l+j) — G(tk + i) 

1 - G(t fc +i) 



k' - k 



l<j<m— 7 

for any threshold (£j)i<j< m +i and letting t = (ij)i<j< m and t' = (ij+i)i<j< m . To prove (j3~Tj) . 
remark that we may assumed that G(ic) = x up to replace (tj)j by (G(tj))j (because G is 
continuous increasing). Next, assume k < k' < m and denote L(r) = Y^=i ^-{Pi — ^r}- By 
definition of a step-down procedure, the probability P [|SD(t)| = fc, |SD(t')| = fc'] is equal to 

P [Vj < fc, L(j) > j, L(k + 1) = fc, Vj, fc + 1 < j < fc', L(j + 1) > j, L(k' + 2) = fc'] 

i) \k> ~ D p t vi - k > m - j ' vj ' & + 1 - j - k '' L{j j ' 

;Pk < h+1 < Pfc+1> — iPk' < *fc'+2 < Pfc'+1> -)Pm] 



Pi, •••,Pfc < ife+1 < Pfc+1, "-jPfc' < *fc'+2 < Pfc'+l, ■■■,Pm , 

k' 



Pi, 

m\ (m — k 
k)\k' - fc 

fc 

Vj <k,^ l{Pi < *,} > J, Vj, fc + 1 < j < fc', l{ Pi < t j+1 } > j - fc 

i=l j=fc+l 

^(^-fc) 10 ^)^* 1 '-'^)^^ a-w 1 -*' 

X P [tk+1 < P(l) < ifc+2, — ,P(k'-k) < ifc'+l] , 
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where we used that the p-values are i.i.d. Simple computations give that 

IP [tfe+l < P(l) < tk+2, — ,P{k'-k) < tk'+l] = IP [P(l) < tk+2 - tk+1, —,P(k'-k) - h'+l ~ h+l] ■ 

This leads to (13111. The cases k < k' = m and k = k' are similar. 



6.3. Proof of (|16p . (|17p and f)18|) . Let us state the following general expression: for 1 < j < 
k < m, 



(32) 



Ho(H) n SD(t)| = j, |SD(t)| = k] 

k-j^lU(i) ^ *i)---!^(fc) < | -ffl = ••• = #j = 0>-ffj+i = ••• = -Hjfc = 1] 
^ * fe (G(ti),...,G(t fe )) 



where (Hi,Ui)i<i<k is a sequence of i.i.d. variables following the unconditional independent 
model (for k hypotheses). Then, applying (132p in the case j ' = k, j = and F\ = 1 will lead to 
(fT6|h (fT7|) and dTHJ) , respectively. To state ([32]) . we use that (Hi,pi)i is a i.i.d. sequence: 

P[|ft (#)nSD(t)| =i,|SD(t)| =k] 

= r) HP[SD(t) = {l,...,k},Hx = ... = Hj = 0,H j+1 = ... = H k = l] 

= Q (7) P[W - ^' S ^ < *4 > ^> V» > fc + > tfc+i, 
Hi = ... = Hj = 0, Hj + i = ... = Hfc = 1] 

= (^Wr^tW < fc, ^ x te < </} > * I #i = - = H j = °> ffi+i = - = #* = !] 

1=1 



x (^](l-G(t* +1 )) m - / -- 



which leads to (132j) by applying (1191) . 

6.4. Proof of Theorem 13.61 We focus on the step-up case, when p G (—1,1) and mo = 1 
(the remaining cases are left to the reader) . Without loss of generality, we may assume that the 
first coordinate correspond to the true null, that is, H = (0, 1). In this context, the FDP takes 
one of the three values: 0, |, 1, according to the location of the tests statistics Yi = Xi + fiHi 
with respect to the critical values Z\ and z 2 ■ From the definition of a step- up procedure, we 
may define the two regions for % G {1,2}, T>i = {(2/1,2/2) £ ^ 2 I FDP (2/1,2/2) = where 
FDP(j/i,2/2) denotes the FDP of SU(t) taken in the p-values p = ($(2/1), $(2/2)) ■ The re- 
gions T>i are represented on Figure E^a). Next, since (Yi,!^) follows the EMN model, we 
may write for i G {1,2}, P(FDP(y x ,Y 2 ) = i/2) = (27^1 - p 2 )" 1 / f^expl-^L^fa - 
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p(y 2 - /u)) 2 - \{y 2 - p) 2 }dy 1 dy 2 = (2ir) 1 / J^. exp{-±(ti 2 + v 2 )} dudv by using the sub- 
stitution u = (yl — p 2 )" 1 (yi — p(j/2 — A 4 )) and v = y 2 — p, and where the resulting in- 



tegration domain Pj is represented on Figure [3{b). Therefore, we obtain P(FDP 



1/2) 



dv, and P(FDP = 1) 



(V27T)- 1 J^exp{-^/2}$ ( (z 2 - pr 



and the final expression results by using the substitution w = &{v) 

( a ) 




FDP = 







FDP = 


1 






Z2 


- V 


FDP = 


1/2 






Z\ 


- V 




Fig 3. Left: T>i. Right: T>i. For i — l(double cross-hatch) and i — 2 (simple cross-hatch). Graph for p < 0. 



6.5. Proof of Theorem\4l\ . For any t G [0, 1], let G(t) = ir t + ir^^t) and G x {t) = ir t + tt 1 . 
First, since Fi is concave, we have for t < t' < 1, that (Fi(t')-Fi (t))/(t'-t) > (l-Fi(i))/(l-i) 
and thus for t < t' , we obtain the inequality 

(33) (G(i') - G(t))/(1 - > (d(0 - d(t))/(l - G x (i)) 

(by convention, the LHS (resp. RHS) of (l33j) is equal to if G(t) = 1 (resp. G\(t) = 1))- From 
expression ([20|) . we obtain that FDR(LSD, Pr no equals 

th ~ ( ( G(tk4-'i) — G(th)\ 
_ _ ,i 

/,:_:.! i=0 yv x ir W /i<j< m - fc 

m m—k 

_ _ ,» 

fc=l i=0 A '" r i V V UU ' W /l<j<m-fc 

m m—k 



nom^V^mt^i^m-uk-l) p ^-Vm-k f( ^G(^ 
< 7r m^P m _i((G(tj))i<j< m _i,fe - 1) ]Tj -j^-^m-k ^(~^Z^^y^) 
T m V]x> m _i((G(£j))i<i<TO-i) fc - 1) V] --^—.T> m -k ( ( tfc 1 +J ^ ) 

fc=l i=0 + ' \ V fc Sl<3<m-k , 
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where the inequality comes from fj33f) and because for a fixed k, the sum over i can be seen as the 
expectation of tk/(k + I) where I is the rejection number of a step-down procedure (point- wise 
nondecreasing in the threshold). Next, considering this time the sum over k as an expectation, 
since G < G\ and since a step-down procedure is point-wise nondecreasing in the threshold, the 
proof is finished by using assumption ((3]) • 

Consider now the case where tj = aj/m and let us prove that this threshold satisfies (plj) . 



For any m > 2 and 1 < k < to, let us denote S, 



m,k 



Em-k _k_T) 
i=0 k+i Ur > 



aj/m 



Letting a m h 



Jli i 



a 



m—k 
m—ak 



(increasing in k) and b. 



'm,k 



m—a m—k 



m m—a 



r (decreasing in k), we 



may prove the following recursion (see the proof below): 

(34) <Sm,fe = a m,k + bm,kSm—l,k- 

Using ([Mjl we propose to state that (t^k satisfies ((21), that is, "fc G {1,...,to} h-> S mj fc is non- 
decreasing" by a recurrence on m > 2: the property is obviously true for to = 2. Assuming the 
property true for m — 1, we obtain for any 2 < < m — 1, 



>m,k 



m—\.k—\i 



Sm,k—1 — Q>m,k ^m,fc— 1 b m ,kSm—l,k &m,fe— l&m— l,k— 1 

^m.k Q'm.k—l {bm.k ^m,fc— l/^m— l,Jfe "I - b m ,k—l(.^m—l,k $ 
^ ("m,fc "I - ^m,fc) (0"m,fc— 1 "I - ^m,fc— 1)> 

" '"~ fc , the quantity a m) k + b m) k is increasing 
1 > Sm m-i) an d the recurrence is 



1- 



because S^-i,*; < 1. Hence, since a mjfc + 6 mifc - - - 
in and <S mi fc — S m ^-i > 0. Also, we obviously have 5 mim 
completed. 



We now finally state (|34j) . Let for 1 < j < m — k, tj 

k 



i and t'- 

1— ak/m J 



l-t 



m—k—j 



+i, so that 



S. 



m,k 



Er 



k + \SD((tj)x<j< m -k) 



TO - |SU((t£)l<j<m-fc)| 



where Eo denotes the expectation with respect to i.i.d. uniform p-values. Hence, denoting t'j 

and 7 = — , we obtain 



/3 + j'7 with ft = 

S m ,k 1 



m— ma— a 



k 



m—ak 

- + -Eo 

TO TO 



|SU((tj)l<i<m-fc)| 
TO - |SU((t'.)i<j< m -fc)| 



, m—k 

-— + —} 
TO TO ^-^ TO — 7 



., , m—k 

1 m — k \ 

- + — E 



m — k 
j 



in 



TO 



TO — J 



TO — — 1 N 



J-l 



(tjf) J 1 ^ / m _A,._j(l — 1 — ij+i), 
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so that equals 



m—X—k 



1 m — k m — k "VA ^ (i' 7 +i) J / m — 1 — k 

+ (/5 + m,7 > 



in 



m 



1 


m — 


k 




m — 


k 




7 




+ 






m 








m 




1 


m — 


k 




m — 


k 




7 




+ 






m 


m 






m 





j=0 

08 + m 7 )E 



m — 1 — j 



*m-l-fc-i(l - C-fc) -j 1 - ^+2) 



(/J + mj) 



and the recursion (134j) is proved. 



m - 1 - |SU((^- +1 )i<j< m _i_fc) 

SjTO— l,fe 
~k~' 



6.6. Proofs for Section Let us first prove ([29]): denote for any < i < m — 1, Ui = 

Eo[(|SU((tj)i4_i<j< TTl )| +i + (where Eo denotes the expectation with respect to i.i.d. uniform 
p- values) and u m = l/(m + 1), u m+ \ = 0, so that no equals the quantity in fl29j) . We may prove 
the following recursion relation: for any < i < m, 

(35) (i + l)m = (1 — (m - i)7) - (m - i)(/3 - 7)^+1. 

Expression (I35p is proved as follows: for i < m, 

|SU((ij)j.fi<j< m )| 



(« + =1 - E 

m- 

=i-E 



|SU((^-)i+l<j<m)| + * + l 



A; 



k=l 



k + i + l 



m — 1 
k 



--1- (m-i)Y^ 



k=l 

m— (i+1) 



(ifc+i) fcv ^m-i-fc(l — t m , 1 — tfc+i+i) 
m — i — 1 



=1 - (m - i) 

k=0 



k+i+l\ k-1 



7 + 



(tk+i) X I / m-i-fc(l — ^m; 1 ~~ tk+i+l) 

m — (i + 1) N 



fc + (i + l) + l 



k 



(*Jfc+(i+l)) 



x ^m-(i+l)-fc(l — *m 5 •••) 1 — *i 



<<fc+(i+l)+IJ • 

Next, we obtain that the solution of the recursion (|35p is given by 



1 - (m - (m-i) x ■■■ x (m-(i + j)) ^ ^ 



3=0 



m — (i + j) 



(i + 1) x ... x (i + jf + 1) 



which leads to u = EJLo M jy h (7) (7 - /?) j and (EHD results. 

To prove Theorem 14.31 we combine (|28|) and (|29p , the latter using m — 1 hypotheses and 
special values for /3 and 7: j3 = 7 = -K^a/m for -Fi(x) = ; /3 = 7 = a/m for iq(x) = 2; 
/3 = -K^a/m + (1 — vro)e and 7 = -KQa/m for -Fi(x) = e. 
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Appendix 

APPENDIX A: USEFUL LEMMAS 
The following lemma is related to the proof of Theorem 2.1 in [12] and to Lemma 8.1 (i) in 

Hi- 

Lemma A.l. Consider a step-up procedure SU(t) using a given threshold t testing m null 
hypotheses with p-values p\ , p m and rejecting k = \SU(t)\ hypotheses. For a given 1 < £ < m, 
denote by k'^ the number of rejections of the step-up procedure of threshold (tj + e)i<j< m -i over 
m — £ hypotheses and using the p-values pi+±, p m - Then we have point-wise 

Vl<i<£, pi<t k Vl<i<l, Pi<t y ^ +e k = k[ e) +l 

Proof. First note that since k is nondecreasing in each coordinate of (pi, ...,p m ), we always 
have k < k %\ + £. Second, since p^ < tf. is equivalent to |{1 < j < m \ pj < tk}\ > k, the 

rejection number of SU(t) can be defined as k = max{/c E {0, l,...,m} | |{1 < j < m | pj < 
tk}\ > k}. Hence, VI < i < £, pi < ty +i is equivalent to |{1 < j < m \ Pj < ty +i }\ > 

£ + \{£+l < j < ni | Pj ^ ty _ lr f\\ which is equivalent to |{1 < j ' < m \ pj < ty _|_^}| ^ i + (^)' 
by definition of fe/« . As a consequence, since A; is a maximum and since k < k ,^ + £, the latter 
is equivalent to k'^ + £ = k. This establishes the second equivalence. The first equivalence easily 
comes from the second equivalence and using that t^ < ty +e because {tk)k is a nondecreasing 
sequence. □ 
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For step-down procedures, we use the next lemma. 

Lemma A. 2. Consider a step-down procedure SD(t) using a given threshold t testing m 
null hypotheses with p-values p\, p m and rejecting k = \SD(t)\ hypotheses. Denote by k^ 
(resp. k'ns) the number of rejections of the step-up procedure of threshold (tj)i<j<m-i (resp. 
fe'+i)i<i<m— i) over m — 1 hypotheses and using the p-values p%, p m . Then we have point- 
wise 

Pi<h Pi < ^ k = k[ x) + 1. 

In the above lemma, we underline that the assertion p x < ty +1 =>• pi < t^ is not true in 
general. 

Proof. Similarly to the step- up case, the rejection number of SD(t) can be defined as k = 
max{k E {0, l,...,m} | \/k' < k, |{1 < j < m \ pj < ty}\ > k'}. Also remark that we always 
have fc^ + 1 > k and, by definition of k, for any j we have pj < t^ 44> pj < % +1 • First prove 

that pi < tjj 4=> = fc^ + 1: using the definitions of k and fc^ we obtain p\ < |{2 < 

j < m | < < k 43- k'^ < k ^ k'^ + 1 = k. Second, we prove p\ > 44> p\ > 1 +1 : 

since we obviously have > k^, we get pi > t% pi > =4> pi > Conversely, if 

Pi > we S et li 1 < i < m I Pi < = |{2 < j < m I Pi < < fe(i) + 1 (by 

definition of kn\), hence fyi) + 1 > k (by definition of k), which implies kn\ = k, thus p\ > , 1 
and finally pi > ir . □ 
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